Global Average Land Temperature Time Series forecasting with a simple Gated Recurrent Units (GRU) Neural Network architecture, using TensorFlow, Keras and Talos.

Time Series forecasting is one of Machine Learning challenges, especially when it comes to weather parameters, as these a subject to multiple processes influence that may hinder a better learning of these sequences of values. In this piece of work we are going to illustrate the power Deep Learning in tackling a typical problem, viz., forecasting the global average land temperature using a 2-layer GRU. The dataset contains monthly temperatures records ranging from 1750, for the longest time series, to 2015.
Just like performing a Grid-/Random-Search with Scikit-Learn, a set of hyperparameters will be optimized using Talos.

##################################################################################################################
##*********                    Moukouba Moutoumounkata, July 2020              ***************                  ##
##                Global Land Average Temperature Time Series Forecasting                                       ##
##                *******************************************************                                       ##
##  NOTE: please, copy and paste the notebook link into http://nbviewer.jupyter.org/ to show the graphics       ##       
##################################################################################################################



#we will try to build a reproductible experience, although without 
#guarantee, due to the extremely random nature of Neural Networks
from numpy.random import seed
seed(1)
from tensorflow import set_random_seed
set_random_seed(2)

import numpy as np
import pandas as pd
from operator import itemgetter
import matplotlib.pyplot as plt
import chart_studio.plotly as py
import plotly.express as px
import plotly.graph_objs as go
import seaborn as sns

from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

# Importing the Keras libraries and packages from TensorFlow
from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (Input, Dense, Dropout, LSTM, Flatten, 
                                     GRU, Bidirectional, TimeDistributed)
from tensorflow.keras.activations import relu, linear
from tensorflow.keras.optimizers import SGD, Adam, RMSprop
from tensorflow.keras.models import model_from_json
from tensorflow.keras import backend as K
import tensorflow as tf

import talos
from talos import scan, Evaluate, Reporting 
from talos.utils import early_stopper, lr_normalizer
from talos.utils.gpu_utils import parallel_gpu_jobs
from talos.utils.recover_best_model import recover_best_model

import warnings
warnings.filterwarnings("ignore")

%matplotlib inline

df = pd.read_csv("/home/moukouba/data_science/python/datasets/globaltemperatures.csv")

Using TensorFlow backend.

df.head()

	dt	landaveragetemperature	landaveragetemperatureuncertainty	landmaxtemperature	landmaxtemperatureuncertainty	landmintemperature	landmintemperatureuncertainty	landandoceanaveragetemperature	landandoceanaveragetemperatureuncertainty
0	1750-01-01	3.034	3.574	NaN	NaN	NaN	NaN	NaN	NaN
1	1750-02-01	3.083	3.702	NaN	NaN	NaN	NaN	NaN	NaN
2	1750-03-01	5.626	3.076	NaN	NaN	NaN	NaN	NaN	NaN
3	1750-04-01	8.490	2.451	NaN	NaN	NaN	NaN	NaN	NaN
4	1750-05-01	11.573	2.072	NaN	NaN	NaN	NaN	NaN	NaN

df.shape

(3192, 9)

df["dt"] = df.dt.apply(pd.to_datetime, errors='coerce')

df1 = df[["dt", "landaveragetemperature", "landaveragetemperatureuncertainty"]].set_index("dt")
df2 = df[["dt", "landmintemperature", "landmintemperatureuncertainty"]].set_index("dt")

print(df1.isnull().sum())
print(df2.isnull().sum())
df1.head(15)

landaveragetemperature               12
landaveragetemperatureuncertainty    12
dtype: int64
landmintemperature               1200
landmintemperatureuncertainty    1200
dtype: int64

	landaveragetemperature	landaveragetemperatureuncertainty
dt
1750-01-01	3.034	3.574
1750-02-01	3.083	3.702
1750-03-01	5.626	3.076
1750-04-01	8.490	2.451
1750-05-01	11.573	2.072
1750-06-01	12.937	1.724
1750-07-01	15.868	1.911
1750-08-01	14.750	2.231
1750-09-01	11.413	2.637
1750-10-01	6.367	2.668
1750-11-01	NaN	NaN
1750-12-01	2.772	2.970
1751-01-01	2.495	3.469
1751-02-01	0.963	3.827
1751-03-01	5.800	3.051

df1.shape

(3192, 2)

df2 = df2.dropna()
df2.head()

	landmintemperature	landmintemperatureuncertainty
dt
1850-01-01	-3.206	2.822
1850-02-01	-2.291	1.623
1850-03-01	-1.905	1.410
1850-04-01	1.018	1.329
1850-05-01	3.811	1.347

As can be seen, the sequence has a few missing values (#12). These will be imputed using the Long Term Mean (LTM) of specific day, but not the LTM of all the series, so as not to affect the range of values that specific day may take.

#We define a function that replaces the missing entries with the long term mean of that specific day of the year
def imputer(df):
    #The actual dates with missing values within all the dataset
    dates_nan = pd.Series([index for index, row in df.iterrows() if row.isnull().any()])
    
    #The specific days of the year with some missing values (without duplicates)
    days_nan = dates_nan.dt.strftime('%m-%d').drop_duplicates()
    
    #The Sets of specific days with missing values
    set_with_missing = [[index for index in df.index if day in str(index)] for day in days_nan]
    
    #Now, we can replace the missings with, say, the mean/mode of each set
    for missing in dates_nan:
        for labels in set_with_missing:
            if missing in labels:
                df.loc[missing,] = df.loc[labels,].mean() 
    return df

df1 = imputer(df1)

df1.head(15)

	landaveragetemperature	landaveragetemperatureuncertainty
dt
1750-01-01	3.034000	3.574000
1750-02-01	3.083000	3.702000
1750-03-01	5.626000	3.076000
1750-04-01	8.490000	2.451000
1750-05-01	11.573000	2.072000
1750-06-01	12.937000	1.724000
1750-07-01	15.868000	1.911000
1750-08-01	14.750000	2.231000
1750-09-01	11.413000	2.637000
1750-10-01	6.367000	2.668000
1750-11-01	5.701539	0.961708
1750-12-01	2.772000	2.970000
1751-01-01	2.495000	3.469000
1751-02-01	0.963000	3.827000
1751-03-01	5.800000	3.051000

Although the main focus of the work is to try and forecast the Land Average Temperature (LAT) only, it has been well worthwhile to pinpoint how global temperatures (average and minimum) have varied over the years, especially since 1900, in the context of Climate Change. That is, a succinct analysis of the Land Minimum Temperature (LMT) has been done accordingly. This has been done through graphical analysis of the series line plots.

#For the ease of readability, the series will be broken into three portions
#x1, x2, x3 = itemgetter(0, 1, 2)(np.array_split(X, 3))
x = np.array_split(df1, 3)

for data in x:
    fig = px.line(data["landaveragetemperature"])
    fig.show()

fig = px.line(df1["landaveragetemperatureuncertainty"])
fig.show()

#z1, z2, z3 = itemgetter(0, 1, 2)(np.array_split(Z, 3))

z = np.array_split(df2, 3)

for data in z:
    fig = px.line(data["landmintemperature"])
    fig.show()

#Subsetting one month, say July, and plot the series' curve
df3 = df1[df1.index.month == 7]
df4 = df2[df2.index.month == 7]

fig = px.line(df3["landaveragetemperature"])
fig.show()

fig = px.line(df4["landmintemperature"])
fig.show()

Carefully looking at the above graphics, we can say that both the LAT and the LMT are indeed rising! It can be noticed that, up to 1930, the LAT was fluctuating about $14°C$; but from 1930 on, the LAT has gone well above this value, exceeding $15°C$ from 1995 onwards. Furthermore, the LMT is also on the rise, with values below $-3°C$ up to 1954; and from this year, the LMT has risen above $-3°C$ and has generally gone above $-2°C$.

Finally, the two last graphics above, which exemplify temperature variations during the month of July, conspicuously show that global warming is a reality, as temperatures, especially the LMT, have been continuously rising since 1900.

Worthy of note, since the World did not have a dense network of observation stations until the last century, these values have been mostly estimated through Re-analysis, and the uncertainty curve shows that the farther we go back in time, the larger the uncertainty in estimation, with values approaching zero in recent decades. Therefore, values up to the end of 1800s should be considered with a grain of salt, as they tend to be inconsistent with the above observations.

Data Preparation

GRUs are Recurrent Neural Network models, and expect three-dimensional input with the shape [samples, timesteps, features]. The data will, thus, be reshaped accordingly. But first, we check for the consistency of the data; then through a certain number of transformations before feeding it to the model.

#Plotting boxplots to check for outliers

trace0 = go.Box(y=df2["landmintemperature"], name="Min_temp")
trace1 = go.Box(y=df1["landaveragetemperature"], name="Avg_temp")

sequences = [trace0, trace1]
fig = go.Figure(sequences)
fig.show()

The above boxplots may be missleading, as they show that the data is free from outliers. But a careful understanding of the variability of temperature with respect to months/seasons is enough to consider each month individually; for example, considering the month of February, we can see that it's not exempt from outliers, as shown in the graphic below. But, as stipulated above, we want to investigate the power of Deep Learing, so will we leave the algorithm to learn and make forecasts.

#Boxplots for the specific month of July
trace0 = go.Box(y=df4["landmintemperature"], name="Min_temp")
trace1 = go.Box(y=df3["landaveragetemperature"], name="Avg_temp")

sequences = [trace0, trace1]
fig = go.Figure(sequences)
fig.show()

#We define a function that transforms the data to the required shape
def convert2matrix(data_arr, n_steps):
    X, y = [], []
    for i in range(len(data_arr) - n_steps):
        d = i + n_steps  
        X.append(data_arr[i:d, ])
        y.append(data_arr[d, ])
    return np.array(X), np.array(y)

#We define a function that splits the data into train, validation and test sets
def data_splitter(df, train_fraction):
    train_size = int(len(df)*train_fraction)
    train, test = df[0:train_size, ], df[train_size:len(df), ]
    return train, test

#Split the data into train, validation and test series
series = df1["landaveragetemperature"].values 

train, test = data_splitter(series, 0.85)
train, val = data_splitter(series, 0.75)
print(len(train), len(val), len(test))

2394 798 479

#Convert dataset into right shape to turn the problem into a supervised... 
#... learning one. We choose n_steps = 36, thus, we use we look back up to... 
#... 3 years (36) consecutive months to be able to forecast the next month

n_steps = 36

X_train, y_train = convert2matrix(train, n_steps)
X_val, y_val = convert2matrix(val, n_steps)
X_test, y_test = convert2matrix(test, n_steps)

#Scale the data
b_scaled = X_train.copy()
b_scaled_val = X_val.copy()
b_scaled_test = X_test.copy()

scaler = MinMaxScaler(feature_range=(0, 1))
x_train = scaler.fit_transform(b_scaled)
x_val = scaler.transform(b_scaled_val)
x_test = scaler.transform(b_scaled_test)

# reshape input to be [samples, time steps, features]
x_train = np.reshape(x_train, (x_train.shape[0], 1, x_train.shape[1]))
x_val = np.reshape(x_val, (x_val.shape[0], 1, x_val.shape[1]))
x_test = np.reshape(x_test, (x_test.shape[0], 1, x_test.shape[1]))

b_scaled_test.shape

(443, 36)

for i in range(5):
    print(b_scaled_test[i], y_test[i])

[ 3.031  4.517  8.294 10.942 13.086 14.155 13.511 11.895  8.511  5.66
  3.681  2.492  3.471  5.702  8.85  11.78  13.876 14.631 14.09  11.862
  9.156  6.544  3.749  2.705  3.456  5.607  8.791 11.414 13.22  14.364
 13.297 12.03   9.339  6.35   3.74   2.679] 2.841
[ 4.517  8.294 10.942 13.086 14.155 13.511 11.895  8.511  5.66   3.681
  2.492  3.471  5.702  8.85  11.78  13.876 14.631 14.09  11.862  9.156
  6.544  3.749  2.705  3.456  5.607  8.791 11.414 13.22  14.364 13.297
 12.03   9.339  6.35   3.74   2.679  2.841] 5.474
[ 8.294 10.942 13.086 14.155 13.511 11.895  8.511  5.66   3.681  2.492
  3.471  5.702  8.85  11.78  13.876 14.631 14.09  11.862  9.156  6.544
  3.749  2.705  3.456  5.607  8.791 11.414 13.22  14.364 13.297 12.03
  9.339  6.35   3.74   2.679  2.841  5.474] 8.455
[10.942 13.086 14.155 13.511 11.895  8.511  5.66   3.681  2.492  3.471
  5.702  8.85  11.78  13.876 14.631 14.09  11.862  9.156  6.544  3.749
  2.705  3.456  5.607  8.791 11.414 13.22  14.364 13.297 12.03   9.339
  6.35   3.74   2.679  2.841  5.474  8.455] 11.199000000000002
[13.086 14.155 13.511 11.895  8.511  5.66   3.681  2.492  3.471  5.702
  8.85  11.78  13.876 14.631 14.09  11.862  9.156  6.544  3.749  2.705
  3.456  5.607  8.791 11.414 13.22  14.364 13.297 12.03   9.339  6.35
  3.74   2.679  2.841  5.474  8.455 11.199] 13.487

Now, we build a basic GRU neural network, with two hidden layers and one dropout-regularization layer. Two metrics (the Mean Absolute Error and the Coefficient of Determination (r_squared)), will be used.

# Setting the dictionary of the hyperparameter to be included in the optimisation process 
p = {'epochs': (20, 150, 5),
     'neurons1': [ 32, 64, 128, 256],
     'neurons2': [32, 64, 128, 256],
     'dropout': (0.1, 0.6, 5),
     'loss': ['mse', 'mae'],
     'activation1':[relu, None,],
     'activation2':[linear, None,],
     'batch_size': [8, 16, 32, 64, 128],
     'optimizer': ['Adam', 'SGD', 'RMSprop']
     }

#Define custom coefficient of determination metric
def r_squared(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true-y_pred)) 
    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) 
    return (1 - SS_res/(SS_tot + K.epsilon()))


#We define a model builder function. We wrap the first hidden layer into a Bidirectional layer 
def model_builder(x_train, y_train, x_val, y_val, params, n_steps = 36, n_features = 1):
    
    tf.keras.backend.clear_session()
    
    model = Sequential([
        Bidirectional(GRU(params['neurons1'], return_sequences=True, 
                          activation=params['activation1'], input_shape=(n_features, n_steps))),
        GRU(params['neurons2'], activation=params['activation1']),
        Dropout(params['dropout']),
        
        Dense(1, activation=params['activation2'])        
    ])
    
    model.compile(optimizer=params['optimizer'], loss=params['loss'], metrics=['mae', r_squared])
    
    history = model.fit(x_train, y_train, epochs=params['epochs'], batch_size=params['batch_size'],  verbose=0, 
                        validation_data=[x_val, y_val], callbacks=[early_stopper(epochs=params['epochs'], 
                                                                                 mode='moderate', monitor='val_loss')])
    
    return history, model

Next, we run the experiments creating a Scan object (scrutinizer) and split the GPU memory in two for two parallel jobs; and for the sake of computational burden, only 1% of the hyperparameter space, randomly downsampled, was used - that is, 480 rounds - and the process took 1 hour and 43 minutes to execute, on a modest GTX-1050 GPU.

parallel_gpu_jobs(0.5)

scrutinizer = talos.Scan(x=x_train, y=y_train, x_val=x_test, y_val=y_test, seed=42, 
                         model=model_builder, experiment_name='time_series__gru_hpo_1', 
                         params=p,fraction_limit=0.01, reduction_metric='val_loss')

  0%|          | 0/480 [00:00<?, ?it/s]

  0%|          | 1/480 [00:19<2:32:28, 19.10s/it]

  0%|          | 2/480 [00:25<2:01:42, 15.28s/it]

  1%|          | 3/480 [00:31<1:39:46, 12.55s/it]

  1%|          | 4/480 [00:45<1:43:09, 13.00s/it]

  1%|          | 5/480 [00:55<1:34:31, 11.94s/it]

  1%|▏         | 6/480 [01:03<1:25:35, 10.83s/it]

  1%|▏         | 7/480 [01:10<1:15:30,  9.58s/it]

  2%|▏         | 8/480 [01:20<1:17:24,  9.84s/it]

  2%|▏         | 9/480 [01:25<1:05:18,  8.32s/it]

  2%|▏         | 10/480 [01:32<1:02:33,  7.99s/it]

  2%|▏         | 11/480 [01:36<54:07,  6.92s/it]  

  2%|▎         | 12/480 [01:47<1:02:17,  7.99s/it]

  3%|▎         | 13/480 [01:53<58:20,  7.50s/it]  

  3%|▎         | 14/480 [01:58<52:14,  6.73s/it]

  3%|▎         | 15/480 [02:04<49:01,  6.32s/it]

  3%|▎         | 16/480 [02:15<1:00:21,  7.80s/it]

  4%|▎         | 17/480 [02:36<1:30:30, 11.73s/it]

  4%|▍         | 18/480 [02:41<1:16:16,  9.91s/it]

  4%|▍         | 19/480 [02:47<1:06:24,  8.64s/it]

  4%|▍         | 20/480 [02:52<57:23,  7.49s/it]  

  4%|▍         | 21/480 [02:57<51:05,  6.68s/it]

  5%|▍         | 22/480 [03:03<50:29,  6.61s/it]

  5%|▍         | 23/480 [03:09<47:42,  6.26s/it]

  5%|▌         | 24/480 [03:30<1:22:07, 10.81s/it]

  5%|▌         | 25/480 [03:46<1:33:13, 12.29s/it]

  5%|▌         | 26/480 [03:51<1:16:08, 10.06s/it]

  6%|▌         | 27/480 [04:00<1:14:32,  9.87s/it]

  6%|▌         | 28/480 [04:25<1:49:08, 14.49s/it]

  6%|▌         | 29/480 [04:59<2:32:04, 20.23s/it]

  6%|▋         | 30/480 [05:03<1:55:11, 15.36s/it]

  6%|▋         | 31/480 [05:10<1:37:14, 12.99s/it]

  7%|▋         | 32/480 [05:25<1:40:50, 13.50s/it]

  7%|▋         | 33/480 [05:50<2:07:16, 17.08s/it]

  7%|▋         | 34/480 [05:59<1:47:57, 14.52s/it]

  7%|▋         | 35/480 [06:12<1:44:05, 14.04s/it]

  8%|▊         | 36/480 [06:19<1:29:02, 12.03s/it]

  8%|▊         | 37/480 [06:24<1:13:23,  9.94s/it]

  8%|▊         | 38/480 [06:29<1:01:05,  8.29s/it]

  8%|▊         | 39/480 [06:34<55:11,  7.51s/it]  

  8%|▊         | 40/480 [06:41<53:48,  7.34s/it]

  9%|▊         | 41/480 [06:54<1:05:02,  8.89s/it]

  9%|▉         | 42/480 [07:12<1:24:11, 11.53s/it]

  9%|▉         | 43/480 [07:17<1:09:46,  9.58s/it]

  9%|▉         | 44/480 [07:52<2:06:08, 17.36s/it]

  9%|▉         | 45/480 [08:01<1:47:41, 14.85s/it]

 10%|▉         | 46/480 [08:08<1:29:27, 12.37s/it]

 10%|▉         | 47/480 [08:19<1:27:26, 12.12s/it]

 10%|█         | 48/480 [08:29<1:21:02, 11.26s/it]

 10%|█         | 49/480 [08:50<1:43:37, 14.43s/it]

 10%|█         | 50/480 [08:55<1:21:55, 11.43s/it]

 11%|█         | 51/480 [09:00<1:07:40,  9.46s/it]

 11%|█         | 52/480 [09:08<1:05:59,  9.25s/it]

 11%|█         | 53/480 [09:17<1:03:32,  8.93s/it]

 11%|█▏        | 54/480 [09:23<58:05,  8.18s/it]  

 11%|█▏        | 55/480 [09:27<49:56,  7.05s/it]

 12%|█▏        | 56/480 [09:36<53:03,  7.51s/it]

 12%|█▏        | 57/480 [09:40<45:53,  6.51s/it]

 12%|█▏        | 58/480 [09:44<40:03,  5.70s/it]

 12%|█▏        | 59/480 [10:00<1:01:19,  8.74s/it]

 12%|█▎        | 60/480 [10:04<50:38,  7.23s/it]  

 13%|█▎        | 61/480 [10:18<1:06:29,  9.52s/it]

 13%|█▎        | 62/480 [10:24<58:03,  8.33s/it]  

 13%|█▎        | 63/480 [10:28<49:26,  7.11s/it]

 13%|█▎        | 64/480 [10:36<51:00,  7.36s/it]

 14%|█▎        | 65/480 [10:45<53:57,  7.80s/it]

 14%|█▍        | 66/480 [10:54<55:43,  8.08s/it]

 14%|█▍        | 67/480 [10:59<49:26,  7.18s/it]

 14%|█▍        | 68/480 [11:04<44:34,  6.49s/it]

 14%|█▍        | 69/480 [11:12<48:19,  7.05s/it]

 15%|█▍        | 70/480 [11:17<43:24,  6.35s/it]

 15%|█▍        | 71/480 [11:22<41:50,  6.14s/it]

 15%|█▌        | 72/480 [11:27<38:48,  5.71s/it]

 15%|█▌        | 73/480 [11:35<43:29,  6.41s/it]

 15%|█▌        | 74/480 [11:45<50:40,  7.49s/it]

 16%|█▌        | 75/480 [11:49<43:35,  6.46s/it]

 16%|█▌        | 76/480 [12:12<1:16:43, 11.40s/it]

 16%|█▌        | 77/480 [12:16<1:01:52,  9.21s/it]

 16%|█▋        | 78/480 [12:24<57:47,  8.63s/it]  

 16%|█▋        | 79/480 [12:29<51:07,  7.65s/it]

 17%|█▋        | 80/480 [12:34<45:25,  6.81s/it]

 17%|█▋        | 81/480 [12:42<47:17,  7.11s/it]

 17%|█▋        | 82/480 [12:46<41:45,  6.30s/it]

 17%|█▋        | 83/480 [12:50<37:07,  5.61s/it]

 18%|█▊        | 84/480 [12:59<43:31,  6.59s/it]

 18%|█▊        | 85/480 [13:07<46:36,  7.08s/it]

 18%|█▊        | 86/480 [13:16<49:17,  7.51s/it]

 18%|█▊        | 87/480 [13:23<49:02,  7.49s/it]

 18%|█▊        | 88/480 [13:30<47:02,  7.20s/it]

 19%|█▊        | 89/480 [13:49<1:10:09, 10.77s/it]

 19%|█▉        | 90/480 [14:19<1:47:37, 16.56s/it]

 19%|█▉        | 91/480 [14:26<1:28:34, 13.66s/it]

 19%|█▉        | 92/480 [14:49<1:46:54, 16.53s/it]

 19%|█▉        | 93/480 [15:02<1:40:28, 15.58s/it]

 20%|█▉        | 94/480 [15:19<1:43:18, 16.06s/it]

 20%|█▉        | 95/480 [15:24<1:20:59, 12.62s/it]

 20%|██        | 96/480 [15:29<1:06:19, 10.36s/it]

 20%|██        | 97/480 [15:38<1:03:57, 10.02s/it]

 20%|██        | 98/480 [15:43<53:20,  8.38s/it]  

 21%|██        | 99/480 [15:55<59:26,  9.36s/it]

 21%|██        | 100/480 [16:37<2:01:43, 19.22s/it]

 21%|██        | 101/480 [16:42<1:34:31, 14.96s/it]

 21%|██▏       | 102/480 [16:54<1:29:13, 14.16s/it]

 21%|██▏       | 103/480 [17:03<1:19:30, 12.65s/it]

 22%|██▏       | 104/480 [17:09<1:06:59, 10.69s/it]

 22%|██▏       | 105/480 [17:46<1:55:37, 18.50s/it]

 22%|██▏       | 106/480 [17:51<1:30:13, 14.47s/it]

 22%|██▏       | 107/480 [17:55<1:11:09, 11.45s/it]

 22%|██▎       | 108/480 [18:05<1:07:58, 10.96s/it]

 23%|██▎       | 109/480 [18:13<1:01:12,  9.90s/it]

 23%|██▎       | 110/480 [18:17<50:35,  8.20s/it]  

 23%|██▎       | 111/480 [18:28<55:38,  9.05s/it]

 23%|██▎       | 112/480 [19:06<1:49:11, 17.80s/it]

 24%|██▎       | 113/480 [19:15<1:32:43, 15.16s/it]

 24%|██▍       | 114/480 [19:24<1:20:23, 13.18s/it]

 24%|██▍       | 115/480 [19:45<1:34:46, 15.58s/it]

 24%|██▍       | 116/480 [19:52<1:19:04, 13.03s/it]

 24%|██▍       | 117/480 [20:01<1:10:35, 11.67s/it]

 25%|██▍       | 118/480 [20:07<1:01:16, 10.16s/it]

 25%|██▍       | 119/480 [20:16<58:40,  9.75s/it]  

 25%|██▌       | 120/480 [20:20<47:28,  7.91s/it]

 25%|██▌       | 121/480 [20:30<52:21,  8.75s/it]

 25%|██▌       | 122/480 [20:54<1:19:34, 13.34s/it]

 26%|██▌       | 123/480 [21:00<1:06:23, 11.16s/it]

 26%|██▌       | 124/480 [21:35<1:48:25, 18.27s/it]

 26%|██▌       | 125/480 [21:47<1:36:27, 16.30s/it]

 26%|██▋       | 126/480 [21:53<1:18:44, 13.35s/it]

 26%|██▋       | 127/480 [22:05<1:14:48, 12.72s/it]

 27%|██▋       | 128/480 [22:13<1:06:05, 11.26s/it]

 27%|██▋       | 129/480 [22:17<54:42,  9.35s/it]  

 27%|██▋       | 130/480 [22:30<1:00:11, 10.32s/it]

 27%|██▋       | 131/480 [22:37<54:05,  9.30s/it]  

 28%|██▊       | 132/480 [22:46<52:38,  9.08s/it]

 28%|██▊       | 133/480 [22:51<46:55,  8.11s/it]

 28%|██▊       | 134/480 [23:04<54:50,  9.51s/it]

 28%|██▊       | 135/480 [23:14<54:38,  9.50s/it]

 28%|██▊       | 136/480 [23:32<1:09:53, 12.19s/it]

 29%|██▊       | 137/480 [24:02<1:39:24, 17.39s/it]

 29%|██▉       | 138/480 [24:09<1:22:37, 14.49s/it]

 29%|██▉       | 139/480 [24:17<1:10:05, 12.33s/it]

 29%|██▉       | 140/480 [25:23<2:42:01, 28.59s/it]

 29%|██▉       | 141/480 [25:27<1:59:39, 21.18s/it]

 30%|██▉       | 142/480 [25:34<1:35:59, 17.04s/it]

 30%|██▉       | 143/480 [25:41<1:17:36, 13.82s/it]

 30%|███       | 144/480 [25:51<1:11:17, 12.73s/it]

 30%|███       | 145/480 [25:57<59:42, 10.69s/it]  

 30%|███       | 146/480 [26:05<55:51, 10.03s/it]

 31%|███       | 147/480 [26:23<1:07:32, 12.17s/it]

 31%|███       | 148/480 [26:42<1:20:09, 14.49s/it]

 31%|███       | 149/480 [27:00<1:24:31, 15.32s/it]

 31%|███▏      | 150/480 [27:03<1:04:51, 11.79s/it]

 31%|███▏      | 151/480 [27:08<53:31,  9.76s/it]  

 32%|███▏      | 152/480 [27:16<50:26,  9.23s/it]

 32%|███▏      | 153/480 [27:23<46:01,  8.45s/it]

 32%|███▏      | 154/480 [27:27<38:14,  7.04s/it]

 32%|███▏      | 155/480 [27:38<45:16,  8.36s/it]

 32%|███▎      | 156/480 [27:42<37:59,  7.03s/it]

 33%|███▎      | 157/480 [27:49<37:48,  7.02s/it]

 33%|███▎      | 158/480 [27:55<35:50,  6.68s/it]

 33%|███▎      | 159/480 [27:59<31:07,  5.82s/it]

 33%|███▎      | 160/480 [28:07<35:07,  6.59s/it]

 34%|███▎      | 161/480 [28:11<30:18,  5.70s/it]

 34%|███▍      | 162/480 [28:19<33:57,  6.41s/it]

 34%|███▍      | 163/480 [28:23<30:40,  5.81s/it]

 34%|███▍      | 164/480 [28:30<32:31,  6.17s/it]

 34%|███▍      | 165/480 [28:45<46:02,  8.77s/it]

 35%|███▍      | 166/480 [29:03<59:56, 11.45s/it]

 35%|███▍      | 167/480 [29:19<1:07:31, 12.94s/it]

 35%|███▌      | 168/480 [29:24<54:49, 10.54s/it]  

 35%|███▌      | 169/480 [29:42<1:06:36, 12.85s/it]

 35%|███▌      | 170/480 [29:47<54:20, 10.52s/it]  

 36%|███▌      | 171/480 [29:52<45:27,  8.83s/it]

 36%|███▌      | 172/480 [30:13<1:02:54, 12.25s/it]

 36%|███▌      | 173/480 [30:16<48:57,  9.57s/it]  

 36%|███▋      | 174/480 [30:34<1:01:55, 12.14s/it]

 36%|███▋      | 175/480 [31:19<1:51:08, 21.86s/it]

 37%|███▋      | 176/480 [31:40<1:49:56, 21.70s/it]

 37%|███▋      | 177/480 [31:44<1:22:25, 16.32s/it]

 37%|███▋      | 178/480 [31:53<1:11:15, 14.16s/it]

 37%|███▋      | 179/480 [32:01<1:02:06, 12.38s/it]

 38%|███▊      | 180/480 [32:07<51:54, 10.38s/it]  

 38%|███▊      | 181/480 [32:16<50:12, 10.08s/it]

 38%|███▊      | 182/480 [32:23<44:48,  9.02s/it]

 38%|███▊      | 183/480 [32:33<46:59,  9.49s/it]

 38%|███▊      | 184/480 [32:39<40:48,  8.27s/it]

 39%|███▊      | 185/480 [32:46<38:38,  7.86s/it]

 39%|███▉      | 186/480 [33:10<1:02:19, 12.72s/it]

 39%|███▉      | 187/480 [33:13<49:12, 10.08s/it]  

 39%|███▉      | 188/480 [33:21<45:11,  9.29s/it]

 39%|███▉      | 189/480 [33:31<45:59,  9.48s/it]

 40%|███▉      | 190/480 [33:46<54:12, 11.21s/it]

 40%|███▉      | 191/480 [33:55<51:09, 10.62s/it]

 40%|████      | 192/480 [34:01<44:17,  9.23s/it]

 40%|████      | 193/480 [34:10<43:36,  9.12s/it]

 40%|████      | 194/480 [34:15<37:05,  7.78s/it]

 41%|████      | 195/480 [34:35<54:52, 11.55s/it]

 41%|████      | 196/480 [34:48<57:05, 12.06s/it]

 41%|████      | 197/480 [34:58<53:00, 11.24s/it]

 41%|████▏     | 198/480 [35:02<42:50,  9.12s/it]

 41%|████▏     | 199/480 [35:22<58:08, 12.41s/it]

 42%|████▏     | 200/480 [35:28<48:18, 10.35s/it]

 42%|████▏     | 201/480 [35:56<1:13:50, 15.88s/it]

 42%|████▏     | 202/480 [36:04<1:01:29, 13.27s/it]

 42%|████▏     | 203/480 [36:34<1:25:29, 18.52s/it]

 42%|████▎     | 204/480 [36:39<1:05:31, 14.24s/it]

 43%|████▎     | 205/480 [36:48<58:22, 12.74s/it]  

 43%|████▎     | 206/480 [36:53<47:29, 10.40s/it]

 43%|████▎     | 207/480 [37:05<49:38, 10.91s/it]

 43%|████▎     | 208/480 [37:13<45:30, 10.04s/it]

 44%|████▎     | 209/480 [37:18<38:29,  8.52s/it]

 44%|████▍     | 210/480 [37:29<41:27,  9.21s/it]

 44%|████▍     | 211/480 [37:47<53:11, 11.86s/it]

 44%|████▍     | 212/480 [37:53<45:39, 10.22s/it]

 44%|████▍     | 213/480 [38:07<50:49, 11.42s/it]

 45%|████▍     | 214/480 [38:15<45:06, 10.17s/it]

 45%|████▍     | 215/480 [38:21<39:19,  8.90s/it]

 45%|████▌     | 216/480 [38:31<40:47,  9.27s/it]

 45%|████▌     | 217/480 [38:40<40:48,  9.31s/it]

 45%|████▌     | 218/480 [38:45<34:29,  7.90s/it]

 46%|████▌     | 219/480 [38:56<38:33,  8.86s/it]

 46%|████▌     | 220/480 [39:08<43:10,  9.96s/it]

 46%|████▌     | 221/480 [39:21<46:24, 10.75s/it]

 46%|████▋     | 222/480 [39:49<1:08:33, 15.95s/it]

 46%|████▋     | 223/480 [39:58<59:25, 13.87s/it]  

 47%|████▋     | 224/480 [40:16<1:04:03, 15.01s/it]

 47%|████▋     | 225/480 [40:28<59:49, 14.08s/it]  

 47%|████▋     | 226/480 [40:36<52:10, 12.33s/it]

 47%|████▋     | 227/480 [40:41<43:15, 10.26s/it]

 48%|████▊     | 228/480 [40:46<35:56,  8.56s/it]

 48%|████▊     | 229/480 [40:53<34:36,  8.27s/it]

 48%|████▊     | 230/480 [40:58<30:14,  7.26s/it]

 48%|████▊     | 231/480 [41:02<26:16,  6.33s/it]

 48%|████▊     | 232/480 [41:13<31:29,  7.62s/it]

 49%|████▊     | 233/480 [41:50<1:07:02, 16.29s/it]

 49%|████▉     | 234/480 [41:54<51:58, 12.68s/it]  

 49%|████▉     | 235/480 [41:59<41:54, 10.26s/it]

 49%|████▉     | 236/480 [42:17<52:09, 12.83s/it]

 49%|████▉     | 237/480 [42:21<40:40, 10.04s/it]

 50%|████▉     | 238/480 [42:46<58:44, 14.56s/it]

 50%|████▉     | 239/480 [43:08<1:07:26, 16.79s/it]

 50%|█████     | 240/480 [43:14<54:18, 13.58s/it]  

 50%|█████     | 241/480 [43:20<45:33, 11.44s/it]

 50%|█████     | 242/480 [43:40<54:41, 13.79s/it]

 51%|█████     | 243/480 [44:03<1:05:31, 16.59s/it]

 51%|█████     | 244/480 [44:10<54:13, 13.79s/it]  

 51%|█████     | 245/480 [44:23<52:31, 13.41s/it]

 51%|█████▏    | 246/480 [44:31<46:28, 11.92s/it]

 51%|█████▏    | 247/480 [44:46<50:02, 12.88s/it]

 52%|█████▏    | 248/480 [44:54<43:20, 11.21s/it]

 52%|█████▏    | 249/480 [44:58<35:34,  9.24s/it]

 52%|█████▏    | 250/480 [45:10<38:19, 10.00s/it]

 52%|█████▏    | 251/480 [45:18<35:52,  9.40s/it]

 52%|█████▎    | 252/480 [45:25<33:10,  8.73s/it]

 53%|█████▎    | 253/480 [45:34<33:40,  8.90s/it]

 53%|█████▎    | 254/480 [45:39<28:23,  7.54s/it]

 53%|█████▎    | 255/480 [45:49<30:43,  8.19s/it]

 53%|█████▎    | 256/480 [45:53<26:31,  7.10s/it]

 54%|█████▎    | 257/480 [46:02<28:32,  7.68s/it]

 54%|█████▍    | 258/480 [46:15<34:17,  9.27s/it]

 54%|█████▍    | 259/480 [46:21<30:59,  8.41s/it]

 54%|█████▍    | 260/480 [46:39<40:33, 11.06s/it]

 54%|█████▍    | 261/480 [46:43<32:57,  9.03s/it]

 55%|█████▍    | 262/480 [47:29<1:13:01, 20.10s/it]

 55%|█████▍    | 263/480 [47:51<1:14:35, 20.62s/it]

 55%|█████▌    | 264/480 [47:59<1:00:28, 16.80s/it]

 55%|█████▌    | 265/480 [48:08<52:36, 14.68s/it]  

 55%|█████▌    | 266/480 [48:14<42:23, 11.89s/it]

 56%|█████▌    | 267/480 [48:19<34:59,  9.86s/it]

 56%|█████▌    | 268/480 [48:28<34:11,  9.67s/it]

 56%|█████▌    | 269/480 [48:32<28:22,  8.07s/it]

 56%|█████▋    | 270/480 [48:50<38:14, 10.93s/it]

 56%|█████▋    | 271/480 [49:28<1:06:18, 19.04s/it]

 57%|█████▋    | 272/480 [49:41<59:21, 17.12s/it]  

 57%|█████▋    | 273/480 [50:05<1:06:19, 19.23s/it]

 57%|█████▋    | 274/480 [50:15<56:12, 16.37s/it]  

 57%|█████▋    | 275/480 [50:19<44:09, 12.92s/it]

 57%|█████▊    | 276/480 [50:31<42:53, 12.61s/it]

 58%|█████▊    | 277/480 [50:57<56:28, 16.69s/it]

 58%|█████▊    | 278/480 [51:26<1:08:18, 20.29s/it]

 58%|█████▊    | 279/480 [51:36<57:17, 17.10s/it]  

 58%|█████▊    | 280/480 [51:48<52:15, 15.68s/it]

 59%|█████▊    | 281/480 [51:53<41:21, 12.47s/it]

 59%|█████▉    | 282/480 [52:00<35:54, 10.88s/it]

 59%|█████▉    | 283/480 [52:16<39:58, 12.18s/it]

 59%|█████▉    | 284/480 [52:29<40:47, 12.49s/it]

 59%|█████▉    | 285/480 [52:35<34:32, 10.63s/it]

 60%|█████▉    | 286/480 [52:40<29:04,  8.99s/it]

 60%|█████▉    | 287/480 [52:46<26:09,  8.13s/it]

 60%|██████    | 288/480 [52:53<24:26,  7.64s/it]

 60%|██████    | 289/480 [53:01<24:34,  7.72s/it]

 60%|██████    | 290/480 [53:06<21:57,  6.93s/it]

 61%|██████    | 291/480 [53:17<25:32,  8.11s/it]

 61%|██████    | 292/480 [53:59<57:45, 18.43s/it]

 61%|██████    | 293/480 [54:10<50:26, 16.18s/it]

 61%|██████▏   | 294/480 [54:14<39:09, 12.63s/it]

 61%|██████▏   | 295/480 [54:38<49:26, 16.04s/it]

 62%|██████▏   | 296/480 [54:51<46:06, 15.04s/it]

 62%|██████▏   | 297/480 [54:55<35:53, 11.77s/it]

 62%|██████▏   | 298/480 [55:17<44:30, 14.67s/it]

 62%|██████▏   | 299/480 [55:59<1:09:19, 22.98s/it]

 62%|██████▎   | 300/480 [56:07<55:20, 18.45s/it]  

 63%|██████▎   | 301/480 [56:19<49:19, 16.53s/it]

 63%|██████▎   | 302/480 [56:26<40:54, 13.79s/it]

 63%|██████▎   | 303/480 [56:37<37:51, 12.83s/it]

 63%|██████▎   | 304/480 [56:42<30:30, 10.40s/it]

 64%|██████▎   | 305/480 [56:56<33:53, 11.62s/it]

 64%|██████▍   | 306/480 [57:07<33:08, 11.43s/it]

 64%|██████▍   | 307/480 [57:17<31:14, 10.84s/it]

 64%|██████▍   | 308/480 [57:20<25:00,  8.72s/it]

 64%|██████▍   | 309/480 [57:29<24:57,  8.76s/it]

 65%|██████▍   | 310/480 [57:56<40:18, 14.23s/it]

 65%|██████▍   | 311/480 [58:00<31:23, 11.15s/it]

 65%|██████▌   | 312/480 [58:09<29:06, 10.40s/it]

 65%|██████▌   | 313/480 [58:16<26:17,  9.44s/it]

 65%|██████▌   | 314/480 [58:26<26:25,  9.55s/it]

 66%|██████▌   | 315/480 [58:32<23:29,  8.54s/it]

 66%|██████▌   | 316/480 [58:56<35:42, 13.07s/it]

 66%|██████▌   | 317/480 [59:01<29:15, 10.77s/it]

 66%|██████▋   | 318/480 [59:06<24:10,  8.96s/it]

 66%|██████▋   | 319/480 [59:19<27:44, 10.34s/it]

 67%|██████▋   | 320/480 [59:59<50:56, 19.11s/it]

 67%|██████▋   | 321/480 [1:00:13<46:20, 17.48s/it]

 67%|██████▋   | 322/480 [1:00:39<52:56, 20.10s/it]

 67%|██████▋   | 323/480 [1:01:05<57:15, 21.88s/it]

 68%|██████▊   | 324/480 [1:01:10<43:54, 16.89s/it]

 68%|██████▊   | 325/480 [1:01:16<35:09, 13.61s/it]

 68%|██████▊   | 326/480 [1:01:21<28:23, 11.06s/it]

 68%|██████▊   | 327/480 [1:01:25<22:46,  8.93s/it]

 68%|██████▊   | 328/480 [1:01:32<20:57,  8.27s/it]

 69%|██████▊   | 329/480 [1:01:38<18:58,  7.54s/it]

 69%|██████▉   | 330/480 [1:01:52<23:58,  9.59s/it]

 69%|██████▉   | 331/480 [1:02:15<33:20, 13.43s/it]

 69%|██████▉   | 332/480 [1:02:32<36:07, 14.65s/it]

 69%|██████▉   | 333/480 [1:02:38<29:10, 11.91s/it]

 70%|██████▉   | 334/480 [1:03:09<42:56, 17.65s/it]

 70%|██████▉   | 335/480 [1:03:15<34:38, 14.34s/it]

 70%|███████   | 336/480 [1:03:25<31:25, 13.09s/it]

 70%|███████   | 337/480 [1:03:37<30:00, 12.59s/it]

 70%|███████   | 338/480 [1:03:47<27:59, 11.83s/it]

 71%|███████   | 339/480 [1:03:52<22:45,  9.68s/it]

 71%|███████   | 340/480 [1:03:59<21:17,  9.12s/it]

 71%|███████   | 341/480 [1:04:04<17:57,  7.75s/it]

 71%|███████▏  | 342/480 [1:04:27<28:17, 12.30s/it]

 71%|███████▏  | 343/480 [1:04:39<28:19, 12.40s/it]

 72%|███████▏  | 344/480 [1:04:54<29:29, 13.01s/it]

 72%|███████▏  | 345/480 [1:05:03<26:20, 11.71s/it]

 72%|███████▏  | 346/480 [1:05:30<36:33, 16.37s/it]

 72%|███████▏  | 347/480 [1:05:44<34:38, 15.63s/it]

 72%|███████▎  | 348/480 [1:06:03<36:33, 16.62s/it]

 73%|███████▎  | 349/480 [1:06:09<29:48, 13.65s/it]

 73%|███████▎  | 350/480 [1:06:20<27:18, 12.60s/it]

 73%|███████▎  | 351/480 [1:06:29<24:59, 11.63s/it]

 73%|███████▎  | 352/480 [1:06:37<22:46, 10.68s/it]

 74%|███████▎  | 353/480 [1:06:59<29:30, 13.94s/it]

 74%|███████▍  | 354/480 [1:07:31<40:46, 19.42s/it]

 74%|███████▍  | 355/480 [1:07:38<32:39, 15.68s/it]

 74%|███████▍  | 356/480 [1:07:51<30:41, 14.85s/it]

 74%|███████▍  | 357/480 [1:07:56<24:42, 12.05s/it]

 75%|███████▍  | 358/480 [1:08:01<19:53,  9.79s/it]

 75%|███████▍  | 359/480 [1:08:16<22:50, 11.33s/it]

 75%|███████▌  | 360/480 [1:08:55<39:20, 19.67s/it]

 75%|███████▌  | 361/480 [1:09:19<41:41, 21.02s/it]

 75%|███████▌  | 362/480 [1:09:29<34:40, 17.63s/it]

 76%|███████▌  | 363/480 [1:09:37<28:31, 14.63s/it]

 76%|███████▌  | 364/480 [1:09:42<22:44, 11.77s/it]

 76%|███████▌  | 365/480 [1:09:46<18:11,  9.49s/it]

 76%|███████▋  | 366/480 [1:09:56<18:40,  9.83s/it]

 76%|███████▋  | 367/480 [1:10:02<16:20,  8.68s/it]

 77%|███████▋  | 368/480 [1:10:16<19:05, 10.23s/it]

 77%|███████▋  | 369/480 [1:10:27<19:21, 10.46s/it]

 77%|███████▋  | 370/480 [1:10:35<17:36,  9.60s/it]

 77%|███████▋  | 371/480 [1:10:39<14:35,  8.03s/it]

 78%|███████▊  | 372/480 [1:10:58<20:16, 11.26s/it]

 78%|███████▊  | 373/480 [1:11:30<30:53, 17.32s/it]

 78%|███████▊  | 374/480 [1:11:57<36:01, 20.39s/it]

 78%|███████▊  | 375/480 [1:12:02<27:31, 15.73s/it]

 78%|███████▊  | 376/480 [1:12:11<23:58, 13.84s/it]

 79%|███████▊  | 377/480 [1:12:17<19:30, 11.36s/it]

 79%|███████▉  | 378/480 [1:12:23<16:23,  9.64s/it]

 79%|███████▉  | 379/480 [1:12:41<20:40, 12.29s/it]

 79%|███████▉  | 380/480 [1:12:51<19:17, 11.57s/it]

 79%|███████▉  | 381/480 [1:12:58<16:56, 10.26s/it]

 80%|███████▉  | 382/480 [1:13:03<14:19,  8.77s/it]

 80%|███████▉  | 383/480 [1:13:09<12:44,  7.88s/it]

 80%|████████  | 384/480 [1:13:15<11:35,  7.24s/it]

 80%|████████  | 385/480 [1:13:24<12:05,  7.63s/it]

 80%|████████  | 386/480 [1:13:27<10:12,  6.52s/it]

 81%|████████  | 387/480 [1:13:35<10:29,  6.77s/it]

 81%|████████  | 388/480 [1:13:40<09:32,  6.22s/it]

 81%|████████  | 389/480 [1:13:45<08:54,  5.87s/it]

 81%|████████▏ | 390/480 [1:13:52<09:31,  6.36s/it]

 81%|████████▏ | 391/480 [1:14:00<10:06,  6.82s/it]

 82%|████████▏ | 392/480 [1:14:35<22:25, 15.29s/it]

 82%|████████▏ | 393/480 [1:14:41<18:13, 12.57s/it]

 82%|████████▏ | 394/480 [1:15:10<24:55, 17.39s/it]

 82%|████████▏ | 395/480 [1:15:28<24:53, 17.57s/it]

 82%|████████▎ | 396/480 [1:15:33<19:21, 13.83s/it]

 83%|████████▎ | 397/480 [1:15:43<17:16, 12.49s/it]

 83%|████████▎ | 398/480 [1:16:04<20:48, 15.23s/it]

 83%|████████▎ | 399/480 [1:16:11<17:17, 12.81s/it]

 83%|████████▎ | 400/480 [1:16:19<15:11, 11.40s/it]

 84%|████████▎ | 401/480 [1:16:26<13:08,  9.98s/it]

 84%|████████▍ | 402/480 [1:16:46<16:44, 12.88s/it]

 84%|████████▍ | 403/480 [1:16:49<12:59, 10.12s/it]

 84%|████████▍ | 404/480 [1:17:03<14:05, 11.13s/it]

 84%|████████▍ | 405/480 [1:17:09<12:10,  9.74s/it]

 85%|████████▍ | 406/480 [1:17:38<18:54, 15.33s/it]

 85%|████████▍ | 407/480 [1:17:41<14:22, 11.81s/it]

 85%|████████▌ | 408/480 [1:17:48<12:22, 10.32s/it]

 85%|████████▌ | 409/480 [1:17:55<11:03,  9.34s/it]

 85%|████████▌ | 410/480 [1:18:01<09:33,  8.20s/it]

 86%|████████▌ | 411/480 [1:18:46<22:06, 19.23s/it]

 86%|████████▌ | 412/480 [1:18:49<16:29, 14.55s/it]

 86%|████████▌ | 413/480 [1:19:02<15:31, 13.91s/it]

 86%|████████▋ | 414/480 [1:19:09<13:11, 11.99s/it]

 86%|████████▋ | 415/480 [1:19:14<10:34,  9.76s/it]

 87%|████████▋ | 416/480 [1:19:18<08:43,  8.17s/it]

 87%|████████▋ | 417/480 [1:19:26<08:19,  7.94s/it]

 87%|████████▋ | 418/480 [1:19:52<13:51, 13.41s/it]

 87%|████████▋ | 419/480 [1:19:56<10:42, 10.54s/it]

 88%|████████▊ | 420/480 [1:20:10<11:32, 11.54s/it]

 88%|████████▊ | 421/480 [1:20:15<09:36,  9.77s/it]

 88%|████████▊ | 422/480 [1:20:20<08:04,  8.35s/it]

 88%|████████▊ | 423/480 [1:20:39<10:53, 11.47s/it]

 88%|████████▊ | 424/480 [1:20:51<10:49, 11.60s/it]

 89%|████████▊ | 425/480 [1:21:29<17:48, 19.42s/it]

 89%|████████▉ | 426/480 [1:21:36<14:19, 15.93s/it]

 89%|████████▉ | 427/480 [1:21:43<11:28, 13.00s/it]

 89%|████████▉ | 428/480 [1:21:53<10:42, 12.36s/it]

 89%|████████▉ | 429/480 [1:22:04<10:02, 11.81s/it]

 90%|████████▉ | 430/480 [1:22:11<08:35, 10.31s/it]

 90%|████████▉ | 431/480 [1:22:30<10:35, 12.97s/it]

 90%|█████████ | 432/480 [1:22:35<08:21, 10.46s/it]

 90%|█████████ | 433/480 [1:22:53<10:07, 12.92s/it]

 90%|█████████ | 434/480 [1:23:00<08:23, 10.95s/it]

 91%|█████████ | 435/480 [1:23:12<08:36, 11.48s/it]

 91%|█████████ | 436/480 [1:23:20<07:39, 10.44s/it]

 91%|█████████ | 437/480 [1:23:27<06:34,  9.18s/it]

 91%|█████████▏| 438/480 [1:23:31<05:23,  7.70s/it]

 91%|█████████▏| 439/480 [1:25:02<22:18, 32.65s/it]

 92%|█████████▏| 440/480 [1:25:37<22:22, 33.56s/it]

 92%|█████████▏| 441/480 [1:25:54<18:25, 28.35s/it]

 92%|█████████▏| 442/480 [1:25:58<13:29, 21.29s/it]

 92%|█████████▏| 443/480 [1:26:03<10:00, 16.24s/it]

 92%|█████████▎| 444/480 [1:26:11<08:15, 13.77s/it]

 93%|█████████▎| 445/480 [1:26:49<12:14, 20.99s/it]

 93%|█████████▎| 446/480 [1:26:55<09:26, 16.67s/it]

 93%|█████████▎| 447/480 [1:27:02<07:30, 13.65s/it]

 93%|█████████▎| 448/480 [1:27:19<07:46, 14.59s/it]

 94%|█████████▎| 449/480 [1:27:30<06:58, 13.50s/it]

 94%|█████████▍| 450/480 [1:27:50<07:42, 15.43s/it]

 94%|█████████▍| 451/480 [1:27:59<06:32, 13.55s/it]

 94%|█████████▍| 452/480 [1:28:08<05:44, 12.29s/it]

 94%|█████████▍| 453/480 [1:28:14<04:40, 10.40s/it]

 95%|█████████▍| 454/480 [1:28:20<03:58,  9.16s/it]

 95%|█████████▍| 455/480 [1:28:29<03:48,  9.16s/it]

 95%|█████████▌| 456/480 [1:28:36<03:20,  8.36s/it]

 95%|█████████▌| 457/480 [1:29:41<09:46, 25.48s/it]

 95%|█████████▌| 458/480 [1:29:50<07:27, 20.32s/it]

 96%|█████████▌| 459/480 [1:30:00<06:01, 17.20s/it]

 96%|█████████▌| 460/480 [1:30:29<06:58, 20.95s/it]

 96%|█████████▌| 461/480 [1:30:34<05:05, 16.09s/it]

 96%|█████████▋| 462/480 [1:30:41<04:01, 13.40s/it]

 96%|█████████▋| 463/480 [1:30:46<03:06, 10.96s/it]

 97%|█████████▋| 464/480 [1:30:51<02:23,  8.98s/it]

 97%|█████████▋| 465/480 [1:31:00<02:17,  9.18s/it]

 97%|█████████▋| 466/480 [1:31:10<02:08,  9.19s/it]

 97%|█████████▋| 467/480 [1:31:27<02:30, 11.60s/it]

 98%|█████████▊| 468/480 [1:32:12<04:20, 21.70s/it]

 98%|█████████▊| 469/480 [1:32:25<03:28, 18.95s/it]

 98%|█████████▊| 470/480 [1:32:28<02:24, 14.40s/it]

 98%|█████████▊| 471/480 [1:32:53<02:37, 17.49s/it]

 98%|█████████▊| 472/480 [1:32:59<01:53, 14.13s/it]

 99%|█████████▊| 473/480 [1:33:04<01:19, 11.32s/it]

 99%|█████████▉| 474/480 [1:33:14<01:05, 10.86s/it]

 99%|█████████▉| 475/480 [1:33:19<00:45,  9.04s/it]

 99%|█████████▉| 476/480 [1:33:34<00:43, 10.89s/it]

 99%|█████████▉| 477/480 [1:33:41<00:29,  9.70s/it]

100%|█████████▉| 478/480 [1:33:53<00:21, 10.55s/it]

100%|█████████▉| 479/480 [1:34:09<00:12, 12.08s/it]

100%|██████████| 480/480 [1:34:26<00:00, 11.81s/it]

scrutinizer.details

experiment_name        time_series__gru_hpo_1
random_method                uniform_mersenne
reduction_method                         None
reduction_interval                         50
reduction_window                           20
reduction_threshold                       0.2
reduction_metric                     val_loss
complete_time                  09/02/20/08:07
x_shape                         (2358, 1, 36)
y_shape                               (2358,)
dtype: object

analyze_object = talos.Analyze(scrutinizer)
analyze_object.data

	start	end	duration	round_epochs	loss	mean_absolute_error	r_squared	val_loss	val_mean_absolute_error	val_r_squared	activation1	activation2	batch_size	dropout	epochs	loss	neurons1	neurons2	optimizer
0	09/02/20-063329	09/02/20-063347	18.524943	55	0.823395	0.823395	0.938036	0.933311	0.933311	0.940276	None	None	128	0.3	124	mae	128	256	RMSprop
1	09/02/20-063348	09/02/20-063354	6.130740	16	1.096337	0.771087	0.943052	0.367303	0.514470	0.977957	None	None	64	0.4	72	mse	32	256	SGD
2	09/02/20-063354	09/02/20-063400	6.011164	11	1.336319	0.858851	0.927582	0.323927	0.484028	0.980157	<function relu at 0x7fbfd5b345f0>	<function linear at 0x7fbfd5b348c0>	32	0.2	46	mse	32	64	SGD
3	09/02/20-063400	09/02/20-063414	13.872175	10	2.056761	1.079151	0.857138	0.418065	0.566391	0.968644	None	<function linear at 0x7fbfd5b348c0>	8	0.3	46	mse	128	32	Adam
4	09/02/20-063415	09/02/20-063424	9.273089	13	1.191160	0.799578	0.932839	0.199008	0.357435	0.987244	<function relu at 0x7fbfd5b345f0>	<function linear at 0x7fbfd5b348c0>	16	0.2	72	mse	32	128	SGD
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
475	09/02/20-080648	09/02/20-080703	14.988790	11	1.124024	1.124024	0.847592	1.144140	1.144140	0.864829	None	None	8	0.5	72	mae	128	32	SGD
476	09/02/20-080703	09/02/20-080710	6.733503	4	2.116007	1.095089	0.880577	0.115561	0.273573	0.992269	<function relu at 0x7fbfd5b345f0>	None	16	0.2	20	mse	256	32	Adam
477	09/02/20-080710	09/02/20-080723	12.293553	10	0.690102	0.690101	0.948978	0.374279	0.374279	0.986697	<function relu at 0x7fbfd5b345f0>	None	32	0.1	46	mae	256	256	Adam
478	09/02/20-080723	09/02/20-080738	15.425810	29	3.563871	1.428061	0.813533	0.124696	0.283167	0.992576	None	<function linear at 0x7fbfd5b348c0>	64	0.5	124	mse	256	32	RMSprop
479	09/02/20-080738	09/02/20-080755	16.834105	23	0.968601	0.704330	0.946285	1.388243	1.126706	0.907356	None	None	16	0.2	72	mse	128	128	SGD

480 rows × 19 columns

# The lowest root mean squared error achieved on validation set 
best_r_squared = analyze_object.high('val_r_squared')
# The lowest mean absolute error achieved on validation set 
best_mae = analyze_object.low('val_mean_absolute_error')

print("The scores for r_squared and mae, on validation set are %.5f and %.5f, respectively."%(best_r_squared, best_mae))

The scores for r_squared and mae, on validation set are 0.99383 and 0.25517, respectively.

# The best models based on respective metrics
best1 = scrutinizer.best_model(metric='val_r_squared', asc=False)
best2 = scrutinizer.best_model(metric='val_mean_absolute_error', asc=False)

# Predicting the Test set results
y_pred_rsq = best1.predict(x_test)
y_pred_mae = best2.predict(x_test)
r2_1 = r2_score(y_test, y_pred_rsq)
r2_2 = r2_score(y_test, y_pred_mae)
print('R-squared for the mae-based metric, on test set, is %.5f .'%(r2_2))

#
y0 = y_test.flatten()
y1 = y_pred_rsq.flatten()
y2 = y_pred_mae.flatten()
results = pd.DataFrame({"y_test":y0, "y_pred_rsq":y1, "y_pred_mae":y2})

results.tail(30)

R-squared for the mae-based metric, on test set, is 0.71681 .

	y_test	y_pred_rsq	y_pred_mae
413	15.003	15.381952	12.523735
414	14.742	14.789345	11.986822
415	13.154	12.893158	10.074577
416	10.256	10.142653	7.558117
417	7.424	7.019255	5.032195
418	4.724	4.404260	3.274255
419	3.732	3.371956	3.134112
420	3.500	3.999748	3.188420
421	6.378	6.059946	4.438654
422	9.589	9.204416	6.779340
423	12.582	12.148882	9.371989
424	14.335	14.478730	11.543098
425	14.873	15.396744	12.496756
426	14.875	14.769712	11.932276
427	13.091	12.866812	10.080663
428	10.330	10.091636	7.532374
429	6.713	7.064516	5.047521
430	4.850	4.400086	3.305976
431	3.881	3.439721	3.140718
432	4.664	4.074718	3.189036
433	6.740	6.355127	4.502402
434	9.313	9.409620	6.885385
435	12.312	12.194939	9.417303
436	14.505	14.334574	11.472509
437	15.051	15.262173	12.364956
438	14.755	14.709978	11.862999
439	12.999	12.870922	10.026600
440	10.801	10.123028	7.515562
441	7.433	7.225534	5.072102
442	5.518	4.655444	3.366595

xpoints = df.iloc[-443:, 0]

trace0 = go.Scatter(x=xpoints, y=y0, name='Actual Values')
trace1 = go.Scatter(x=xpoints, y=y1, name='Predicted (RSQ)')
trace2 = go.Scatter(x=xpoints, y=y2, name='Predicted (MAE)')

data = [trace0, trace1, trace2]

layout=go.Layout(title="Actual and Predicted Temperatures", xaxis={'title':'Year'}, yaxis={'title':'Temprature'})
fig = go.Figure(data=data, layout=layout)
fig.show()

# We can now save the best model for further use or deployment

# Get the best model index based on the highest 'validation r_squared' 
model_id = analyze_object.data[['val_r_squared']].idxmax()[0]

# Clear any previous TensorFlow session.
tf.keras.backend.clear_session()

# Load the model parameters from the scanner.
model = model_from_json(scrutinizer.saved_models[model_id])
model.set_weights(scrutinizer.saved_weights[model_id])
model.summary()
model.save('./avg_land_temp_best_model.h5')

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bidirectional (Bidirectional multiple                  450048    
_________________________________________________________________
gru_1 (GRU)                  multiple                  52320     
_________________________________________________________________
dropout (Dropout)            multiple                  0         
_________________________________________________________________
dense (Dense)                multiple                  33        
=================================================================
Total params: 502,401
Trainable params: 502,401
Non-trainable params: 0
_________________________________________________________________

Conclusion:

As can be seen, a GRU achieves a mind-blowing Coefficiencient of Determination (r_squared) of 99% on completely new data (test set). The curve (red line), of the predicted values perfectly mimics the actual values curve (blue line), almost completely shading it. Also, the r_squared-based model, by far, outperforms the one based on Mean Absolute Error.

To implement a GRU on the above time series, the number of steps was chosen through trials, and 36 has shown to yield a good performance. Unfortunately, Talos does not have a mechanism that can permit to use an sklearn data processing pipeline in order to incorporate n_steps in the hyperparameter grid so as to automatically search its best value.